Guided parallelized stochastic gradient descent for delay compensation

Stochastic gradient descent (SGD) algorithm and its variations have been effectively used to optimize neural network models. However, with the rapid growth of big data deep learning, SGD is no longer most suitable choice due natural behavior sequential optimization error function. This has led development parallel algorithms, such as asynchronous (ASGD) synchronous (SSGD) train networks. it introduces a high variance delay in parameter (weight) update. We address this our proposed try minimize impact. employed guided (gSGD) that encourages consistent examples steer convergence by compensating unpredictable deviation caused delay. Its rate also similar A/SSGD, however, some additional (parallel) processing required compensate for The experimental results demonstrate approach able mitigate impact quality classification accuracy. SSGD clearly outperforms even achieves accuracy close benchmark datasets.